feat: collect benchmark data into 'benchmarks' branch#762
Merged
Conversation
Adds two Nuke targets and a docs-site-compatible aggregator so the matrix Benchmarks-* artifacts produced by the build workflow can be turned into the data.js / limited-data.js files consumed by the documentation site (mirroring the aweXpect setup). - PublishBenchmarkReport: downloads all Benchmarks-* artifacts of the current run, parses *-report-full-compressed.json files, and commits updated data.js / limited-data.js (last 50 commits) to the 'benchmarks' branch via the GitHub Contents API. The branch is auto-created from main on first run. - BenchmarkSeedHistory: one-shot tool that walks past successful build.yml runs on main, downloads each run's Benchmarks-* artifacts, fetches commit metadata from the workflow run head_commit, and seeds the benchmark data files. Idempotent - already-recorded SHAs are skipped. - PageBenchmarkReportGenerator: chart key incorporates BenchmarkDotNet Parameters (e.g. 'Method (N=1)'), so each [Params] combination becomes its own chart. Includes Mockolate, Moq, NSubstitute, FakeItEasy, TUnitMocks, Imposter as datasets. - build.yml: new publish-benchmark-report job that runs after the matrix benchmarks job on push to main only, with contents: write.
There was a problem hiding this comment.
Pull request overview
Adds automation to aggregate BenchmarkDotNet JSON artifacts from CI into docs-site-consumable data.js / limited-data.js, publishing them to a dedicated benchmarks branch (and providing a one-shot history seeding target).
Changes:
- Introduces
PageBenchmarkReportGeneratorto aggregate BenchmarkDotNet JSON reports intowindow.BENCHMARK_DATAJS payloads (including parameterized chart keys). - Adds Nuke targets
PublishBenchmarkReportandBenchmarkSeedHistoryplus GitHub Contents/Actions API helpers inBuildExtensions. - Extends
build.ymlwith a post-benchmarks job to publish the aggregated benchmark data to thebenchmarksbranch.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| Pipeline/PageBenchmarkReportGenerator.cs | New aggregator that converts BenchmarkDotNet JSON into docs-site data.js / limited-data.js. |
| Pipeline/BuildExtensions.cs | Adds GitHub API helpers for branch creation, file read/write, and workflow run listing. |
| Pipeline/Build.Benchmarks.cs | Adds targets to publish benchmark data for current run and seed from historical runs. |
| .nuke/build.schema.json | Exposes new targets/parameter to Nuke tooling schema. |
| .github/workflows/build.yml | Adds a publish job that runs on main after benchmark matrix completes. |
🚀 Benchmark ResultsDetails
Details
Details
Details
Details
Details
|
Remove EnsureBranchExistsAsync, ParseOrCreate, BenchmarkSeedHistory and their supporting types (ListSuccessfulRunsAsync, WorkflowRunInfo, BenchmarkSeedRunLimit parameter). The 'benchmarks' branch and data files have already been seeded, so PublishBenchmarkReport can now assume both exist - matching the aweXpect surface. If the branch or data file ever goes missing, PublishBenchmarkReport will fail loudly (NRE on dataFile.Content / NotSupportedException on the missing window.BENCHMARK_DATA prefix), which is the same behaviour as aweXpect's BenchmarkReport target.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Comment on lines
+158
to
+180
| BuildExtensions.GithubFile dataFile = | ||
| await BuildExtensions.ReadBranchFileAsync(BenchmarkDataPath, BenchmarkBranch, GithubToken); | ||
| BuildExtensions.GithubFile limitedFile = | ||
| await BuildExtensions.ReadBranchFileAsync(BenchmarkLimitedDataPath, BenchmarkBranch, GithubToken); | ||
|
|
||
| (string updated, string limited) = PageBenchmarkReportGenerator.Append( | ||
| commitInfo, | ||
| dataFile.Content, | ||
| benchmarkReports, | ||
| BenchmarkLimit); | ||
|
|
||
| if (string.IsNullOrWhiteSpace(updated)) | ||
| { | ||
| Log.Information("No changes to publish (commit already recorded)."); | ||
| return; | ||
| } | ||
|
|
||
| string commitMessage = | ||
| $"Update benchmark for {commitInfo.Sha.Substring(0, 8)}: {commitInfo.Message} by {commitInfo.Author}"; | ||
| await BuildExtensions.WriteBranchFileAsync(BenchmarkDataPath, BenchmarkBranch, commitMessage, updated, | ||
| dataFile.Sha, GithubToken); | ||
| await BuildExtensions.WriteBranchFileAsync(BenchmarkLimitedDataPath, BenchmarkBranch, commitMessage, | ||
| limited, limitedFile.Sha, GithubToken); |
Comment on lines
+201
to
+235
| Output[] lines = GitTasks.Git("log -1").ToArray(); | ||
| string commitId = null, author = null, date = null, message = null; | ||
| foreach (string line in lines.Select(x => x.Text)) | ||
| { | ||
| if (commitId == null && line.StartsWith("commit ")) | ||
| { | ||
| commitId = line.Substring("commit ".Length).Substring(0, 40); | ||
| continue; | ||
| } | ||
|
|
||
| if (author == null && line.StartsWith("Author: ")) | ||
| { | ||
| author = line.Substring("Author: ".Length); | ||
| int index = author.IndexOf(" <", StringComparison.Ordinal); | ||
| if (index > 0) | ||
| { | ||
| author = author.Substring(0, index); | ||
| } | ||
|
|
||
| continue; | ||
| } | ||
|
|
||
| if (date == null && line.StartsWith("Date: ")) | ||
| { | ||
| date = line.Substring("Date: ".Length); | ||
| continue; | ||
| } | ||
|
|
||
| if (commitId != null && author != null && date != null && !string.IsNullOrWhiteSpace(line)) | ||
| { | ||
| message = line.Trim(); | ||
| break; | ||
| } | ||
| } | ||
|
|
| $"{RepositoryApiBaseUrl}/contents/{path}?ref={Uri.EscapeDataString(branch)}"); | ||
| if (!response.IsSuccessStatusCode) | ||
| { | ||
| return null; |
Comment on lines
+196
to
+214
| public static async Task WriteBranchFileAsync(string path, string branch, string commitMessage, string content, | ||
| string existingSha, string githubToken) | ||
| { | ||
| using HttpClient client = CreateGithubClient(githubToken); | ||
| GithubUpdateFile body = new(commitMessage, Base64Encode(content), existingSha, branch); | ||
| HttpResponseMessage response = await client.PutAsync( | ||
| $"{RepositoryApiBaseUrl}/contents/{path}", | ||
| new StringContent(JsonSerializer.Serialize(body), Encoding.UTF8, "application/json")); | ||
| if (response.IsSuccessStatusCode) | ||
| { | ||
| Log.Information("Updated {Path} on branch '{Branch}'.", path, branch); | ||
| } | ||
| else | ||
| { | ||
| string responseContent = await response.Content.ReadAsStringAsync(); | ||
| throw new InvalidOperationException( | ||
| $"Could not update '{path}' on branch '{branch}': {responseContent}"); | ||
| } | ||
| } |
Comment on lines
+47
to
+66
| PageReportData pageReport = | ||
| JsonSerializer.Deserialize<PageReportData>(currentFileContent.Substring(FilePrefix.Length)); | ||
|
|
||
| if (pageReport.Values.Any(r => r.Commits.Any(c => c.Sha == commitInfo.Sha))) | ||
| { | ||
| Log.Warning( | ||
| "The benchmark already has data for {Sha}: {Message} by {Author} on {Date}", | ||
| commitInfo.Sha, commitInfo.Message, commitInfo.Author, commitInfo.Date); | ||
| return (null, null); | ||
| } | ||
|
|
||
| Log.Debug( | ||
| "Updating benchmark report for {Sha}: {Message} by {Author} on {Date}", | ||
| commitInfo.Sha, commitInfo.Message, commitInfo.Author, commitInfo.Date); | ||
|
|
||
| foreach (string benchmarkReportContent in benchmarkReportsContents) | ||
| { | ||
| BenchmarkReport benchmarkReport = JsonSerializer.Deserialize<BenchmarkReport>(benchmarkReportContent); | ||
| if (!pageReport.Append(commitInfo, benchmarkReport)) | ||
| { |
Comment on lines
+277
to
+303
| public Benchmark[] Benchmarks { get; init; } | ||
|
|
||
| public class Benchmark | ||
| { | ||
| public string Type { get; init; } | ||
| public string Method { get; init; } | ||
| public string Parameters { get; init; } | ||
| public BenchmarkStatistics Statistics { get; init; } | ||
| public BenchmarkMetrics[] Metrics { get; init; } | ||
| } | ||
|
|
||
| public class BenchmarkStatistics | ||
| { | ||
| public double Mean { get; init; } | ||
| } | ||
|
|
||
| public class BenchmarkMetrics | ||
| { | ||
| public double Value { get; init; } | ||
| public BenchmarkMetricDescriptor Descriptor { get; init; } | ||
| } | ||
|
|
||
| public class BenchmarkMetricDescriptor | ||
| { | ||
| public string Id { get; init; } | ||
| public string DisplayName { get; init; } | ||
| public string Unit { get; init; } |
Comment on lines
+106
to
+130
| publish-benchmark-report: | ||
| name: "Publish Benchmark Report" | ||
| needs: [ benchmarks ] | ||
| if: github.ref == 'refs/heads/main' | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| actions: read | ||
| contents: write | ||
| env: | ||
| DOTNET_NOLOGO: true | ||
| steps: | ||
| - uses: actions/checkout@v6 | ||
| with: | ||
| fetch-depth: 0 | ||
| - name: Setup .NET SDKs | ||
| uses: actions/setup-dotnet@v5 | ||
| with: | ||
| dotnet-version: | | ||
| 10.0.x | ||
| - name: Publish benchmark report | ||
| run: ./build.sh PublishBenchmarkReport | ||
| env: | ||
| GithubToken: ${{ secrets.GITHUB_TOKEN }} | ||
| WorkflowRunId: ${{ github.run_id }} | ||
|
|
github-actions Bot
added a commit
that referenced
this pull request
May 3, 2026
…hmarks' branch (#762) by Valentin Breuß
github-actions Bot
added a commit
that referenced
this pull request
May 3, 2026
…hmarks' branch (#762) by Valentin Breuß
|
This is addressed in release v3.2.0. |
This was referenced May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two Nuke targets and a docs-site-compatible aggregator so the matrix Benchmarks-* artifacts produced by the build workflow can be turned into the data.js / limited-data.js files consumed by the documentation site (mirroring the aweXpect setup).